

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Ceph messenger DPDKStack &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/doctools.js"></script>
        <script src="../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="序列化（编码、解码）" href="../encoding/" />
    <link rel="prev" title="为 Ceph 写作文档" href="../documenting/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../internals/">Ceph 内幕</a></li>
      <li class="breadcrumb-item active">Ceph messenger DPDKStack</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../_sources/dev/dpdk.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../balancer-design/">Ceph 如何均衡（读写、容量）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../blkin/">Tracing Ceph With LTTng</a></li>
<li class="toctree-l2"><a class="reference internal" href="../blkin/#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a></li>
<li class="toctree-l2"><a class="reference internal" href="../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../context/">CephContext</a></li>
<li class="toctree-l2"><a class="reference internal" href="../continuous-integration/">Continuous Integration Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crush-msr/">CRUSH MSR (Multi-step Retry)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Ceph messenger DPDKStack</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#compiling-dpdkstack">Compiling DPDKStack</a></li>
<li class="toctree-l3"><a class="reference internal" href="#setting-the-dpdk-network-adapter">Setting the DPDK Network Adapter</a></li>
<li class="toctree-l3"><a class="reference internal" href="#configuring-osd-dpdkstack">Configuring OSD DPDKStack</a></li>
<li class="toctree-l3"><a class="reference internal" href="#debug-and-optimization">Debug and Optimization</a></li>
<li class="toctree-l3"><a class="reference internal" href="#status-and-future-work">Status and Future Work</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../iana/">IANA 号</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/">Testing changes to the Linux Kernel CephFS driver</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-one-build-the-kernel">Step One: build the kernel</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-two-create-a-vm">Step Two: create a VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-three-networking-the-vm">Step Three: Networking the VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libcephfs_proxy/">Design of the libcephfs proxy</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/">What is a mempool?</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/#some-common-mempools-that-we-can-track">Some common mempools that we can track</a></li>
<li class="toctree-l2"><a class="reference internal" href="../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crimson/">Crimson developer documentation</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="ceph-messenger-dpdkstack">
<h1>Ceph messenger DPDKStack<a class="headerlink" href="#ceph-messenger-dpdkstack" title="Permalink to this heading"></a></h1>
<section id="compiling-dpdkstack">
<h2>Compiling DPDKStack<a class="headerlink" href="#compiling-dpdkstack" title="Permalink to this heading"></a></h2>
<p>Ceph dpdkstack is not compiled by default. Therefore, you need to recompile and
enable the DPDKstack component.
Optionally install <code class="docutils literal notranslate"><span class="pre">dpdk-devel</span></code> or <code class="docutils literal notranslate"><span class="pre">dpdk-dev</span></code> on distros with precompiled DPDK packages, and compile</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">do_cmake.sh<span class="w"> </span>-DWITH_DPDK<span class="o">=</span>ON</span>
</pre></div></div></section>
<section id="setting-the-dpdk-network-adapter">
<h2>Setting the DPDK Network Adapter<a class="headerlink" href="#setting-the-dpdk-network-adapter" title="Permalink to this heading"></a></h2>
<p>Most mainstream NICs support SR-IOV and can be virtualized into multiple VF NICs.
Each OSD uses some dedicated NICs through DPDK. The mon, mgr and client use the PF NICs
through the POSIX protocol stack.</p>
<p>Load the driver on which DPDK depends:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt2:before {
  content: "# ";
}
</style><span class="prompt2">modprobe<span class="w"> </span>vfio</span>
<span class="prompt2">modprobe<span class="w"> </span>vfio_pci</span>
</pre></div></div><p>Configure Hugepage by editing <code class="docutils literal notranslate"><span class="pre">/etc/sysctl.conf</span></code></p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">vm</span><span class="o">.</span><span class="n">nr_hugepages</span> <span class="o">=</span> <span class="n">xxx</span>
</pre></div>
</div>
<p>Configure the number of VFs based on the number of OSDs:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2"><span class="nb">echo</span><span class="w"> </span><span class="nv">$numvfs</span><span class="w"> </span>&gt;<span class="w"> </span>/sys/class/net/<span class="nv">$port</span>/device/sriov_numvfs</span>
</pre></div></div><p>Binding NICs to DPDK Applications:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2">dpdk-devbind.py<span class="w"> </span>-b<span class="w"> </span>vfio-pci<span class="w"> </span><span class="m">0000</span>:xx:yy.z</span>
</pre></div></div></section>
<section id="configuring-osd-dpdkstack">
<h2>Configuring OSD DPDKStack<a class="headerlink" href="#configuring-osd-dpdkstack" title="Permalink to this heading"></a></h2>
<p>By default, the DPDK RTE initialization process requires the root privileges
for accessing various resources in system. To grant the root access to
the <code class="docutils literal notranslate"><span class="pre">ceph</span></code> user:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt2">usermod<span class="w"> </span>-G<span class="w"> </span>root<span class="w"> </span>ceph</span>
</pre></div></div><p>The OSD selects the NICs using <code class="docutils literal notranslate"><span class="pre">ms_dpdk_devs_allowlist</span></code>:</p>
<ol class="arabic">
<li><p>Configure a single NIC.</p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="na">ms_dpdk_devs_allowlist</span><span class="o">=</span><span class="s">-a 0000:7d:010</span>
</pre></div>
</div>
<p>or</p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="na">ms_dpdk_devs_allowlist</span><span class="o">=</span><span class="s">--allow=0000:7d:010</span>
</pre></div>
</div>
</li>
<li><p>Configure the Bond Network Adapter</p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="na">ms_dpdk_devs_allowlist</span><span class="o">=</span><span class="s">--allow=0000:7d:01.0 --allow=0000:7d:02.6 --vdev=net_bonding0,mode=2,slave=0000:7d:01.0,slave=0000:7d:02.6</span>
</pre></div>
</div>
</li>
</ol>
<p>DPDK-related configuration items are as follows:</p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="k">[osd]</span>
<span class="na">ms_type</span><span class="o">=</span><span class="s">async+dpdk</span>
<span class="na">ms_async_op_threads</span><span class="o">=</span><span class="s">1</span>

<span class="na">ms_dpdk_port_id</span><span class="o">=</span><span class="s">0</span>
<span class="na">ms_dpdk_gateway_ipv4_addr</span><span class="o">=</span><span class="s">172.19.36.1</span>
<span class="na">ms_dpdk_netmask_ipv4_addr</span><span class="o">=</span><span class="s">255.255.255.0</span>
<span class="na">ms_dpdk_hugepages</span><span class="o">=</span><span class="s">/dev/hugepages</span>
<span class="na">ms_dpdk_hw_flow_control</span><span class="o">=</span><span class="s">false</span>
<span class="na">ms_dpdk_lro</span><span class="o">=</span><span class="s">false</span>
<span class="na">ms_dpdk_enable_tso</span><span class="o">=</span><span class="s">false</span>
<span class="na">ms_dpdk_hw_queue_weight</span><span class="o">=</span><span class="s">1</span>
<span class="na">ms_dpdk_memory_channel</span><span class="o">=</span><span class="s">2</span>
<span class="na">ms_dpdk_debug_allow_loopback</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">true</span>

<span class="k">[osd.x]</span>
<span class="na">ms_dpdk_coremask</span><span class="o">=</span><span class="s">0xf0</span>
<span class="na">ms_dpdk_host_ipv4_addr</span><span class="o">=</span><span class="s">172.19.36.51</span>
<span class="na">public_addr</span><span class="o">=</span><span class="s">172.19.36.51</span>
<span class="na">cluster_addr</span><span class="o">=</span><span class="s">172.19.36.51</span>
<span class="na">ms_dpdk_devs_allowlist</span><span class="o">=</span><span class="s">--allow=0000:7d:01.1</span>
</pre></div>
</div>
</section>
<section id="debug-and-optimization">
<h2>Debug and Optimization<a class="headerlink" href="#debug-and-optimization" title="Permalink to this heading"></a></h2>
<p>Locate faults based on logs and adjust logs to a proper level:</p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="na">debug_dpdk</span><span class="o">=</span><span class="s">xx</span>
<span class="na">debug_ms</span><span class="o">=</span><span class="s">xx</span>
</pre></div>
</div>
<p>if the log contains a large number of retransmit messages,reduce the value of ms_dpdk_tcp_wmem.</p>
<p>Run the perf dump command to view DPDKStack statistics:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>daemon<span class="w"> </span>osd.<span class="nv">$i</span><span class="w"> </span>perf<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>dpdk</span>
</pre></div></div><p>if the <code class="docutils literal notranslate"><span class="pre">dpdk_device_receive_nombuf_errors</span></code> keeps increasing, check whether the
throttling exceeds the limit:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>daemon<span class="w"> </span>osd.<span class="nv">$i</span><span class="w"> </span>perf<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>throttle-osd_client<span class="w"> </span>-A<span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">&quot;get_or_fail_fail&quot;</span></span>
<span class="prompt1">ceph<span class="w"> </span>daemon<span class="w"> </span>osd.<span class="nv">$i</span><span class="w"> </span>perf<span class="w"> </span>dump<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>throttle-msgr_dispatch_throttler<span class="w"> </span>-A<span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">&quot;get_or_fail_fail&quot;</span></span>
</pre></div></div><p>if the throttling exceeds the threshold, increase the throttling threshold or
disable the throttling.</p>
<p>Check whether the network adapter is faulty or abnormal.Run the following
command to obtain the network adapter status and statistics:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>daemon<span class="w"> </span>osd.<span class="nv">$i</span><span class="w"> </span>show_pmd_stats</span>
<span class="prompt1">ceph<span class="w"> </span>daemon<span class="w"> </span>osd.<span class="nv">$i</span><span class="w"> </span>show_pmd_xstats</span>
</pre></div></div><p>Some DPDK versions (eg. dpdk-20.11-3.e18.aarch64) or NIC TSOs are abnormal,
try disabling tso:</p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="na">ms_dpdk_enable_tso</span><span class="o">=</span><span class="s">false</span>
</pre></div>
</div>
<p>if VF NICs support multiple queues, more NIC queues can be allocated to a
single core to improve performance:</p>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="na">ms_dpdk_hw_queues_per_qp</span><span class="o">=</span><span class="s">4</span>
</pre></div>
</div>
</section>
<section id="status-and-future-work">
<h2>Status and Future Work<a class="headerlink" href="#status-and-future-work" title="Permalink to this heading"></a></h2>
<p>Compared with POSIX Stack, in the multi-concurrency test, DPDKStack has the same
4K random write performance, 8K random write performance is improved by 28%, and
1 MB packets are unstable. In the single-latency test,the 4K and 8K random write
latency is reduced by 15% (the lower the latency is, the better).</p>
<p>At a high level, our future work plan is:</p>
<blockquote>
<div><p>OSD multiple network support (public network and cluster network)
The public and cluster network adapters can be configured.When connecting or
listening,the public or cluster network adapters can be selected based on the
IP address.During msgr-work initialization,initialize both the public and cluster
network adapters and create two DPDKQueuePairs.</p>
</div></blockquote>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../documenting/" class="btn btn-neutral float-left" title="为 Ceph 写作文档" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../encoding/" class="btn btn-neutral float-right" title="序列化（编码、解码）" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>